Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

Just adding a note here as ran into this when working with a GP subclass for the error quantification. This call:

@radka-j - adding here as it might be addressed by the upcoming changes to this API? But if not happy to open a new issue to look at this. An option could also be to revisit having a central registry class to handle this uniformly.

@sgreenbury I don't think we should ever use the GaussianProcess or GaussianProcessCorrelated classes so this to me feels like correct behaviour. If we want a GP class for an RBF + constant kernel we should add that specifically to the registry.

It was the GP context (passing a create_gp_subclass instance to AutoEmulate) I ran into this issue and a workaround might have been to also look at GP_REGISTRY since this maintains a registry of all GPs including the created subclasses.

But thinking more about it, it affects any subclass used by AutoEmulate currently if reinitialize is called, e.g. in the advanced tutorial:

class SimpleFNN(PyTorchBackend):
    ...
ae = AutoEmulate(x, y, models=[SimpleFNN])
ae.fit_from_reinitialized(x, y)

since SimpleFNN is constructed at runtime the class is not found in the lists of emulators.

I think if the emulator becomes the entity that does the refitting in this PR then a global emulator registry including all custom subclasses would not be needed for this but might still be useful?

The lodget, trace and max_eigval plots in the AL documentation look wrong after the refactor here (they barely change). I started trying to figure out what's happening and have a sense that the predicted uncertainty is narrowed when using a GP wrapped inside a TransformedEmulator (even without any transforms) vs just a GP. I need to investigate this more formally but we need to understand what's happening before we can merge this.

I don't know what the issue is yet but my previous comment about the uncertainty from TransformedEmulator being narrower was wrong. I was comparing GP vs TransformedEmulator with GP using different learning rates. Once the same learning rate was used they look visually identical.

It might be related to whether posterior_predictive=True is being passed to the reinitialized GP when within the TransformedEmulator?

For example, on main in the dim reduction tutorial:
https://github.com/alan-turing-institute/autoemulate/blob/6d4a92fdcb2614b5dee5f907855e7003503c0910/docs/tutorials/emulation/02_dim_reduction.ipynb

em = ae.fit_from_reinitialized(x[train_idx], y[train_idx])

has:

print(em.model.posterior_predictive)
False

though the original AutoEmulate initialization having posterior_predictive=True.

Thank you for checking! In this case the posterior_predictive is correctly set to True after the emulator is re-initialized each time.

@sgreenbury I'm also not sure if you saw my previous comment but the uncertainty output from TransformedEmulator seems to be fine.

@sgreenbury I tried running the AL notebook using a GP wrapped inside a TransformedEmulator but calling emulator.fit instead of fit_from_reinitialized as originally implemented and the results look the very similar to the current docs. So it looks like the issue comes from re-initializing the emulator. Given the GP is refitting 1 data point at a time, this might be a case where calling fit with the hyperparameters fixed might actually make sense.

I therefore decided to revert this change and leave AL as is in this PR (only updating typing). We can separately decide whether to leave the associated issue (#757) open to revisit at some later point or close.